{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ">原文转载自「刘悦的技术博客」[https://v3u.cn/a_id_97](https://v3u.cn/a_id_97)\n",
    "\n",
    "最近无论是面试还是笔试，有一个高频问题始终阴魂不散，那就是给一个大文件，至少超过 10g, 在内存有限的情况下（低于 2g），该以什么姿势读它？\n",
    "\n",
    "所有人都知道，用 python 读文件有一套” 标准流程 “：\n",
    "\n",
    "```py\n",
    "def retrun_count(fname):\n",
    "    \"\"\"计算文件有多少行\n",
    "    \"\"\"\n",
    "    count = 0\n",
    "    with open(fname) as file:\n",
    "        for line in file:\n",
    "            count += 1\n",
    "    return count\n",
    "\n",
    "```\n",
    "\n",
    "为什么这种文件读取方式会成为标准？这是因为它有两个好处：\n",
    "\n",
    "with 上下文管理器会自动关闭打开的文件描述符  \n",
    "在迭代文件对象时，内容是一行一行返回的，不会占用太多内存\n",
    "\n",
    "但这套标准做法并非没有缺点。如果被读取的文件里，根本就没有任何换行符，那么上面的第二个好处就不成立了。当代码执行到 for line in file 时，line 将会变成一个非常巨大的字符串对象，消耗掉非常可观的内存。\n",
    "\n",
    "如果有一个 5GB 大的文件 big_file.txt，它里面装满了随机字符串。只不过它存储内容的方式稍有不同，所有的文本都被放在了同一行里\n",
    "\n",
    "如果我们继续使用前面的 return_count 函数去统计这个大文件行数。那么在一台 pc 上，这个过程会足足花掉 65 秒，并在执行过程中吃掉机器 2GB 内存\n",
    "\n",
    "为了解决这个问题，我们需要暂时把这个 “标准做法” 放到一边，使用更底层的 file.read() 方法。与直接循环迭代文件对象不同，每次调用 file.read(chunk_size) 会直接返回从当前位置往后读取 chunk_size 大小的文件内容，不必等待任何换行符出现。\n",
    "\n",
    "所以，如果使用 file.read() 方法，我们的函数可以改写成这样:\n",
    "\n",
    "```py\n",
    "def return_count_v2(fname):\n",
    "\n",
    "    count = 0\n",
    "    block_size = 1024 * 8\n",
    "    with open(fname) as fp:\n",
    "        while True:\n",
    "            chunk = fp.read(block_size)\n",
    "            # 当文件没有更多内容时，read 调用将会返回空字符串 ''\n",
    "            if not chunk:\n",
    "                break\n",
    "            count += 1\n",
    "    return count\n",
    "\n",
    "```\n",
    "\n",
    "在新函数中，我们使用了一个 while 循环来读取文件内容，每次最多读取 8kb 大小，这样可以避免之前需要拼接一个巨大字符串的过程，把内存占用降低非常多。\n",
    "\n",
    "利用生成器解耦代码\n",
    "\n",
    "假如我们在讨论的不是 Python，而是其他编程语言。那么可以说上面的代码已经很好了。但是如果你认真分析一下 return_count_v2 函数，你会发现在循环体内部，存在着两个独立的逻辑：数据生成（read 调用与 chunk 判断） 与 数据消费。而这两个独立逻辑被耦合在了一起。\n",
    "\n",
    "为了提升复用能力，我们可以定义一个新的 chunked_file_reader 生成器函数，由它来负责所有与 “数据生成” 相关的逻辑。这样 return_count_v3 里面的主循环就只需要负责计数即可。\n",
    "\n",
    "```py\n",
    "def chunked_file_reader(fp, block_size=1024 * 8):\n",
    "    \"\"\"生成器函数：分块读取文件内容\n",
    "    \"\"\"\n",
    "    while True:\n",
    "        chunk = fp.read(block_size)\n",
    "        # 当文件没有更多内容时，read 调用将会返回空字符串 ''\n",
    "        if not chunk:\n",
    "            break\n",
    "        yield chunk\n",
    "\n",
    "\n",
    "def return_count_v3(fname):\n",
    "    count = 0\n",
    "    with open(fname) as fp:\n",
    "        for chunk in chunked_file_reader(fp):\n",
    "            count += 1\n",
    "    return count\n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "进行到这一步，代码似乎已经没有优化的空间了，但其实不然。iter(iterable) 是一个用来构造迭代器的内建函数，但它还有一个更少人知道的用法。当我们使用 iter(callable, sentinel) 的方式调用它时，会返回一个特殊的对象，迭代它将不断产生可调用对象 callable 的调用结果，直到结果为 setinel 时，迭代终止。\n",
    "\n",
    "```py\n",
    "def chunked_file_reader(file, block_size=1024 * 8):\n",
    "    \"\"\"生成器函数：分块读取文件内容，使用 iter 函数\n",
    "    \"\"\"\n",
    "    # 首先使用 partial(fp.read, block_size) 构造一个新的无需参数的函数\n",
    "    # 循环将不断返回 fp.read(block_size) 调用结果，直到其为 '' 时终止\n",
    "    for chunk in iter(partial(file.read, block_size), ''):\n",
    "        yield chunk\n",
    "\n",
    "```\n",
    "\n",
    "最后只需要两行代码，就构造出了一个可复用的分块读取方法，和一开始的” 标准流程 “按行读取 2GB 内存 / 耗时 65 秒 相比，使用生成器的版本只需要 7MB 内存 / 12 秒就能完成计算。效率提升了接近 4 倍，内存占用更是不到原来的 1%，简直完美。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
